The invention relates to an optical correlator, for comparing images. Such devices can be used for optical recognition, for example for fingerprint recognition.
Several designs for optical correlators have been proposed. For example, Binary Phase-Only Matched Filter (BPOMF) based designs have been produced for a variety of applications. Correlation in a BPOMF is obtained by multiplying together the Fourier transform of the reference and input functions (r and s). This product is then Fourier transformed again to give the final correlation of r and s. In order to form the product in an optical system the input is displayed on one spatial light modulator and Fourier transformed with a lens. The reference r is Fourier transformed off-line and the result is converted to suit the type of spatial light modulator. The Fourier transform of s then passes through the spatial light modulator containing the Fourier transform of r giving the product. This is where the weakness of the system lies as the Fourier transform of s must be scaled and aligned with the reference to within one pixel at the spatial light modulator. Hence optical design and alignment of opto-mechanics are critical and very difficult to implement outside the laboratory. Another disadvantage of these systems is that the spatial light modulators (SLMs) used are too slow, difficult and expensive to obtain, or both.
Spatial light modulators based on ferroelectric liquid crystals are very fast and offer a potentially cheap technology for optical systems. However, they are limited by their binary modulation, i.e. by the ability of each cell only to display two states. Joint transform correlators using such devices are known from Guibert et al, xe2x80x9cOn-board optical transform correlator for road sign recognitionxe2x80x9d, Optical Engineering, Volume 34 (1995) page 135. This paper describes the use of ferroelectric liquid crystals with an optically addressed spatial light modulator.
However, such a correlator is difficult to construct and there are similar problems in optical design and mechanics as there are with the BPOMF. Also, optically addressed spatial light modulator (OASLM) technology has yet to become reliable and cannot deliver comparable performance to an electrically addressed silicon backplane spatial light modulator.
In a joint transform correlator (JTC), the input and reference images are displayed side-by-side on a display. In a so-called 1/f JTC, as described in J. L. Horner and C. K. Makekau xe2x80x98Two-focal-length optical correlatorxe2x80x99, Applied Optics 28 (12) 1989, pp 2358-2367, the display is illuminated by collimated laser light and the side-by-side images are Fourier-transformed using a lens to form the joint power spectrum (JPS) as an intermediate image. Then, the intermediate image is non-linearly-processed and Fourier-transformed again, using the same or a different lens. The result gives a measure of the correlation between the input and reference images. In this prior-art JTC, the processing on the JPS was not designed to reduce the zero order light in the correlation plane. This was mostly due to the choice of display technology which restricts the modulation of the light to amplitude only. This device was also slow and could not be used to achieve high-speed correlation.
There is thus a need for an improved optical correlation method and correlator to alleviate these difficulties.
According to the first aspect of the invention there is provided a method of optical correlation including the steps of modulating an input image and a reference image with a phase-encoded chequerboard pattern, displaying the modulated images side-by-side on a spatial light modulator, and performing a joint transform correlation on the displayed image.
The joint transform correlation is preferably performed by obtaining the joint power spectrum (JPS) corresponding to the Fourier transform of the input and reference images, and then obtaining a correlation image corresponding to the Fourier transform of the JPS. The correlation image contains information about the correlation between input and reference images.
The correlation is preferably performed by shining collimated light onto the spatial light modulator, forming an intermediate image of the spatial light modulator through a lens, recording and processing the intermediate image (JPS) and displaying the result on a spatial light modulator, shining collimated light onto the latter spatial light modulator, and recording a resulting correlation image of the spatial light modulator through a lens.
The advantage of carrying out the phase-encoding in a chequerboard pattern is that the collimated light passing straight through adjacent areas of the spatial light modulator, i.e. the zero-order light, destructively interferes. This greatly reduces the central zero-order spot of the image, and so helps reduce the contrast that the camera must record.
It is highly advantageous for the method to be a two-pass method, using only one spatial light modulator (SLM), lens and camera; in other words the SLMs and lenses mentioned are the same in each pass. Such a method comprises the steps of firstly displaying the reference and input images on the spatial light modulator and recording the intermediate image with a camera, secondly processing the intermediate image and thirdly displaying the processed intermediate image on the same spatial light modulator, and finally recording the correlation image with the camera to give an indication of the correlation between the input and reference images.
In alternative embodiments, two separate sets of modulators, lenses and cameras are used: this could operate slightly faster but would be more complex and expensive.
In one arrangement, the spatial light modulator (SLM) is a transmissive SLM, so that the light is transmitted through the SLM, through the lens and is then recorded by a camera located approximately one focal length behind the lens.
An alternative arrangement is to use a reflective spatial light modulator. In this arrangement reflected light is passed in the same way through the lens, reflected by the modulator and recorded by a camera.
Preferably the recorded image corresponds to the Fourier transform of the image displayed on the spatial light modulator. This is achieved by using collimated light and the arrangement of the camera one focal length behind the lens. Carrying out a Fourier transform twice on the side-by-side reference and input images gives a correlation image containing information about the correlation between the images. Of course, the Fourier transform will not be exact, since the camera can only record the intensity of the recorded light, not the phase, and background noise will always be present.
According to a second aspect of the invention there is provided a method of optical correlation for obtaining a correlation image corresponding to the correlation between an input and a reference image, including displaying the input and reference images on a spatial light modulator, and performing a joint transform correlation by shining collimated light onto the spatial light modulator, forming an intermediate image of the spatial light modulator through a lens, recording the intermediate image electronically as a plurality of pixels, binarising the intermediate image by thresholding each pixel using an average value of the surrounding pixels, displaying the binarised intermediate image on a spatial light modulator, shining collimated light onto the spatial light modulator, the aforesaid correlation image being the image through a lens of the intermediate image on the spatial light modulator. The intermediate image corresponds to the joint power spectrum of the reference and input images.
The method of binarising an image using the average value of the surrounding pixels is known, in crude edge detection methods, but has not previously been applied to joint transform correlation. The use of this method greatly enhances the correlation image by suppressing the zero order.
Preferably, the method of binarising the intermediate image is to threshold each pixel based on the mean value of each of the eight surrounding pixels. In other words, using pij to indicate the value of the intermediate image pixel at (i,j), the binarised result Pxe2x80x2ij is given by       p    ij    xe2x80x2    =            1      ⁢              xe2x80x83            ⁢      if      ⁢              xe2x80x83            ⁢              p        ij               greater than                             1          /          8                ⁢                  (                                    p                                                i                  -                  1                                ,                                  j                  -                  1                                                      +                          p                                                i                  -                  1                                ,                j                                      +                          p                                                i                  -                  1                                ,                                  j                  +                  1                                                      +                          p                              i                ,                                  j                  -                  1                                                      +                          p                              i                ,                                  j                  +                  1                                                      +                          p                                                i                  +                  1                                ,                                  j                  -                  1                                                      +                          p                                                i                  +                  1                                ,                j                                      +                          p                                                i                  +                  1                                ,                                  j                  +                  1                                                              )                    -              1        ⁢                  xe2x80x83                ⁢                  otherwise          .                    
Preferably the method according to the second aspect is used in combination with modulating the input and reference images with a phase-encoded chequerboard pattern, as described above.
The second aspect may also encompass the other possibilities described above with reference to the first aspect.
According to a third aspect of the invention there is provided a joint transform correlator comprising an electrically addressed ferroelectric liquid crystal spatial light modulator (FLC SLM) for modulating collimated input light, a lens, a camera for capturing modulated light after it has passed through the lens and producing an signal corresponding thereto and a control means for recording the captured image and for addressing the ferroelectric liquid-crystal spatial light modulator, wherein the correlator is adapted to operate in a two-pass process to produce a correlation image from an input image and a reference image. Such correlators have not previously been realised, as far as the applicants are aware. It has not previously been known how such a system could produce correlation images in view of the binary phase nature of the display and without overloading the camera.
The ferroelectric liquid crystal modulator is preferably a binarising liquid crystal modulator with a plurality of pixels each of which can switch between two states outputting light in antiphase with respect to each other. The switching in such liquid crystal modulators is caused by applying an electrical signal to the pixel, and can be very fast: 20 kHz is easily possible. In embodiments, a transmissive ferroelectric liquid crystal spatial light modulator is used. The correlated light is passed directly through the spatial light modulator, the lens and then arrives at the camera where it is recorded.
The spatial light modulator may be a silicon back plane (reflective) SLM to allow a very small correlator, with a length of about 10 cm, compared to 50 cm in prior art arrangements. The optical components used may be made of plastics, for cheapness.
In alternative embodiments, a reflective ferroelectric liquid crystal spatial light modulator is used. The layout here is slightly difference, with a source of correlated light on the same side of the spatial light modulator as the lens. The principle is the same, in that collimated light is reflected by the spatial light modulator, passes through the ens and then arrives at the camera where it is recorded. Reflective ferroelectric devices with very small pixels are available, so these devices can be used to make a very compact and fast joint transform correlator.
Preferably, the control means is adapted to phase-encode the input image and the reference image using a chequerboard pattern, to display the images on the spatial light modulator, to take the recorded image, to process it and to display the processed image on the spatial light modulator, and in turn to output the correlation image.
Preferably, the control means is further adapted to binarise the intermediate image by using a 3xc3x973 convolution kernel. This method thresholds each pixel based on the mean value of each of the eight surrounding pixels. In other words, using Pij to indicate the value of the intermediate image pixel at (i,j), the binarised result Pxe2x80x2ij is given by       p    ij    xe2x80x2    =            1      ⁢              xe2x80x83            ⁢      if      ⁢              xe2x80x83            ⁢              p        ij               greater than                             1          /          8                ⁢                  (                                    p                                                i                  -                  1                                ,                                  j                  -                  1                                                      +                          p                                                i                  -                  1                                ,                j                                      +                          p                                                i                  -                  1                                ,                                  j                  +                  1                                                      +                          p                              i                ,                                  j                  -                  1                                                      +                          p                              i                ,                                  j                  +                  1                                                      +                          p                                                i                  +                  1                                ,                                  j                  -                  1                                                      +                          p                                                i                  +                  1                                ,                j                                      +                          p                                                i                  +                  1                                ,                                  j                  +                  1                                                              )                    -              1        ⁢                  xe2x80x83                ⁢                  otherwise          .                    
Such a binarised spectrum gives good sharp correlation peaks and reduces zero order. This binarisation technique produces a roughly edge-enhanced binary version of the intermediate image. There is no zero order in the Fourier transform of the phase encoded input to swamp the camera. The non-linear process ensures that the binary phase intermediate image after thresholding has approximately equal numbers of +1 and xe2x88x921 points. Hence, when the second Fourier transform is taken there is virtually no zero order (known as DC terms) in the correlation output which means that the detection of the correlation peaks with the CCD is easier and less susceptible to spurious noise peaks.
The camera can be any device that converts the pattern of light falling onto it into an electrical signal. In particular, a charge-coupled device (CCD) may be used, or alternatively a custom silicon photodiode array which can be designed as a smart detector array which also carries out the binarisation process.
The spatial light modulator, lens and camera are preferably arranged so that the image recorded by the camera corresponds to the Fourier transform of the image displayed by the spatial light modulator. For this, the camera is arranged at the focal point of the lens, whereby all the collimated light passing straight through the spatial light modulator ends up at a central spot of the camera. Broadly speaking, light that is diffracted at the spatial light modulator may end up elsewhere on the camera; the shorter the periodicity at the spatial light modulator the greater the angle of deflection of the first order diffraction pattern and hence the further the light ends up from the central spot. This conversion of periodicity at the spatial light modulator to different positions at the camera is a Fourier transform.
In order to display the input and reference images, they are first converted into a binary image of +1 and xe2x88x921 states. Then the modulation in a phase-inversion chequerboard pattern is carried out. The images are multiplied by a chequerboard pattern of xe2x88x921s and 1s to give an encoded input.
Further preferably, the chequerboard corresponds to pixels of the spatial light modulator; in other words, alternate pixels are inverted. The strong first-order diffraction peak is thereby moved outwards as far as possible.
The camera preferably has an aperture of dimensions such that it covers substantially all of the first order diffraction pattern of the image displayed on the spatial light modulator. When the chequerboard pattern corresponds to individual pixels then the strong first-order diffraction peaks are at the corners of the diffraction pattern because no smaller periodicity can be displayed. In order that these strong peaks do not overload the camera it may be advantageous to arrange the camera aperture to be slightly smaller than the size of the first-order diffraction pattern, to exclude these peaks.
Preferably, the control means is adapted to phase-encode the input image and the reference image, to display them on the spatial light modulator, to take the recorded image, to process it and to display the processed image on the spatial light modulator, and in turn to output the correlation image.
The camera can be any device that converts the pattern of light falling onto it into an electrical signal. In particular, a charge-coupled device (CCD) may be used, or photo-diode array.
In a particularly advantageous embodiment, a non-linear CMOS camera is used to capture the Fourier transform of the image. This has two advantages. Firstly, the camera can be made to image over five decades of intensity instead of the 256 gray-scale levels of a CCD camera. Since this more accurately matches the optical distribution of the Fourier spectrum, more information can be picked up. Even with binarisation, this increase the information content of the Fourier transform. The correlation peaks are much stronger and there is more flexibility in how the spectrum can be processed. Secondly, a CMOS detection array can operate at high speed. 2000 frames per second or more are possible. This is much faster than a CCD could deliver.
In embodiments a xe2x80x9csmart pixelxe2x80x9d array integrating the detector, frame grabber and computer could be used. The thresholding would be implemented on the smart pixel array itself, for example in hardware. This approach could readily be combined with a CMOS camera.
According to a fourth aspect of the invention there is provided a method of industrial inspection of products passing a video camera, comprising the steps of recording images of the individual products passing the video camera, displaying pairs of recorded images on a correlator as described above, and outputting the correlation between the pair of the recorded images as a measure of disturbances in the products.
This method allows the detection of defects even when there is no information about the object to be inspected. The current frame and previous frame are synchronised with the progress of objects through the system (in this example, roadsigns). If the sequence does not change, then the output correlations remain from frame to frame. When a change occurs (in this example a rotated roadsign), then the correlation between frames is interrupted. Moreover, the cycle of distortion can be detected by looking at the sequence of disturbances about the first detected defect. Even gradual distortions in the object can be picked up by correlating over multiples of frames to look for small changes. Most importantly, the whole process is done without ever knowing anything about the object being inspected.